Repository navigation
Conversation
A Databricks App to run the assessment, review and edit the table mapping, and migrate tables from a web UI, deployed with `databricks labs ucx deploy-app`. - Backend (FastAPI) in databricks.labs.ucx.app, reusing the CLI's code paths - Frontend (Next.js static export) in app-ui/, built into the wheel - On-behalf-of-user access, with the app's service principal for the Jobs API - Selected migration workflows run sequentially as one chained job run - deploy-app deploys the released wheel as an asset bundle - CI job that builds the frontend and checks the wheel contains the app Co-authored-by: Isaac <no-reply@databricks.com>
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #5000 +/- ##
==========================================
+ Coverage 87.79% 88.02% +0.23%
==========================================
Files 123 135 +12
Lines 17595 18084 +489
Branches 3717 3790 +73
==========================================
+ Hits 15448 15919 +471
- Misses 1458 1466 +8
- Partials 689 699 +10 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the UCX App, a Databricks App that runs the main table migration steps from a web UI, and a new
databricks labs ucx deploy-appcommand to deploy it.workflowsassessmentworkflow, embeds the assessment dashboardensure-assessment-runcreate-table-mapping+ editingmapping.csvmigrate-tables, hiveserde-in-place and CTAS, sequentiallymigrate-tablesAn Active runs window on every page loads the UCX runs in progress when the app opens (including runs started from the CLI or the Jobs UI), polls every 10 s, and keeps finished runs with their final state for 5 minutes. Workflows always run asynchronously.
deploy-appdownloads the released UCX wheel of the installed version, verifies its sha256 against the package index, deploys it as an asset bundle (databricks bundle deploy+bundle run), and grants the app's service principalCAN_MANAGE_RUNon the UCX jobs.Docs:
docs/ucx/docs/reference/app.mdx, thedeploy-appcommand reference, and a developer section incontributing.mdx.Design decisions
src/databricks/labs/ucx/app/), served by gunicorn with uvicorn workers, and a Next.js static export (app-ui/), the same stack as other internal Databricks Apps. The backend reusesWorkspaceContextand the CLI's code paths (TableMapping,DeployedWorkflows,VerifyProgressTracking).(requested by <user>), and the app logs who started what.--run-as-sp trueuses the SP for everything;deploy-appthen also grants itCAN_EDITon the install folder.jobs.submit), with arun_job_taskper workflow, eachdepends_onthe previous one. The chain survives app restarts, and a failure stops the workflows after it.databricks labs install ucxinstalls from the source archive, which doesn't contain the built frontend.deploy-apptherefore deploys the released wheel from PyPI (orPIP_INDEX_URL), or a local wheel with--wheel. The wheel shipsapp/bundle/requirements.txt, generated fromuv.lockwith hashes, so the app runs exactly the locked dependency set in pip hash-checking mode.pyproject.tomlgets a new optional extraapp(fastapi, uvicorn, gunicorn), so CLI and job installs don't grow. The CLI does not import FastAPI.Supply chain
uv.lock. The registry URLs inuv.lockare rewritten to pypi.org the same way asmake lock-dependencies.httpxis added to thepytestgroup for FastAPI'sTestClient.pnpm-lock.yamland a frozen install. This cooldown is applied by hand to direct dependencies only; pnpm 9.15.9 can't enforceminimumReleaseAge, so transitive dependencies are not held back.7.1.0-beta. The internal npm proxy currently serves no stable TypeScript release (5.x and 6.x return 404; only 7.1 pre-releases are listed). This should move to a stable release once it's available.push.ymlare pinned by SHA:actions/setup-nodev7.1.0 andpnpm/action-setupv6.1.1, both verified commits at least 7 days old. JFrog auth configures npm withignore-scripts=true.Release requirement
The release workflow was removed in #4739, so releases are built outside this repo. Release builds must run
make build(it now runsmake app-uiandmake app-requirementsfirst, and needs Node 26 + pnpm 9.15.9). Otherwise the released wheel has no app anddeploy-apprefuses it. The newappCI job runsmake buildand fails if the wheel is missing the app.Testing
tests/unit/app/(API, auth, deployer, CLI).make lintchecks are clean: black, ruff, mypy, pylint 10.00/10.tsc --noEmitandnext buildpass. Clicked through all pages in a browser against the real FastAPI app with canned data: mapping edits and duplicate-destination validation, chained run tracking, no console errors.make buildproduces a wheel with the frontend and hashed requirements. Staged the app from that wheel withAppDeployer, installed it in a clean venv with--require-hashes, and started it with the exactapp.yamlgunicorn command: pages return 200, and the API returns 403 without a user token.yarn build, withonBrokenLinks: throw).tests/integration/app/) have not passed yet. The test workspace's IP access list blocked the machine I ran them from. One partial run in another workspace confirmed that chainedrun_job_tasks are accepted in a submitted run and that a failed step stops the next one (UPSTREAM_FAILED). Its first no-op child job failed for a reason I couldn't see; the test now reports child run errors. This PR's acceptance run should covertest_mapping_roundtrip,test_status_dashboard_and_runsandtest_chained_migration_run.test_deploy_app_end_to_endis skipped unlessUCX_APP_WHEELpoints to a wheel frommake build. It needs a manual run (or a follow-up to build the wheel inacceptance.yml).Not yet verified in a workspace
workspace.workspaceuser scope covers reading and writing the install folder (config.yml,state.json,mapping.csv).CAN_EDITgrant on the install folder in--run-as-spmode.deploy-appagainst PyPI. It's unit-tested with mocked responses only, and some package proxies may not support the JSON API.This pull request and its description were written by Isaac.